Skip to content

Add functionality for export of latency logs via telemetry #608

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 72 commits into
base: telemetry
Choose a base branch
from
Open

Add functionality for export of latency logs via telemetry #608

wants to merge 72 commits into from

Conversation

saishreeeee
Copy link
Collaborator

@saishreeeee saishreeeee commented Jun 19, 2025 •
edited
Loading

What type of PR is this?

  • Refactor
  • Feature
  • Bug Fix
  • Other

Description

Add functionality for export of latency logs via telemetry

Example latency log:
{
"frontend_log_event_id": "30565427-1299-4fbc-bec0-25378cfe912b",
"context": {
"client_context": {
"timestamp_millis": 1750592858108,
"user_agent": "PyDatabricksSqlConnector/4.0.4"
}
},
"entry": {
"sql_driver_log": {
"session_id": "01f04f5e-b1a8-1dd1-962a-e40efd092bdd",
"system_configuration": {
"driver_version": "4.0.4",
"os_name": "Darwin",
"os_version": "24.5.0",
"os_arch": "arm64",
"runtime_name": "Python 3.13.3",
"runtime_version": "3.13.3",
"runtime_vendor": "CPython",
"driver_name": "Databricks SQL Python Connector",
"char_set_encoding": "utf-8",
"locale_name": "en_US"
},
"driver_connection_params": {
"http_path": "/sql/1.0/warehouses/864004c1b3961382",
"mode": "THRIFT",
"host_info": {
"host_url": "e2-dogfood.staging.cloud.databricks.com",
"port": 443
},
"auth_mech": "PAT"
},
"sql_statement_id": "01f04f5e-b1d6-1a1e-afe4-e99e1ccb8805",
"sql_operation": {
"statement_type": "sql",
"is_compressed": true,
"execution_result": "inline_arrow",
"retry_count": 0
},
"operation_latency_ms": 518
}
}
}

How is this tested?

  • Unit tests (will add unit tests)
  • E2E Tests
  • Manually
  • N/A

Related Tickets & Documents

PECOBLR-554

saishreeeee added 30 commits June 17, 2025 21:35
@saishreeeee
added functionality for export of failure logs
65a75f4 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
changed logger.error to logger.debug in exc.py
5305308 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
Fix telemetry loss during Python shutdown
ba83c33 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
unit tests for export_failure_log
131db92 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
try-catch blocks to make telemetry failures non-blocking for connecto…
3abc40d 
…r operations

Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
removed redundant try/catch blocks, added try/catch block to initiali…
ffa4787 
…ze and get telemetry client

Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
skip null fields in telemetry request
cc077f3 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
removed dup import, renamed func, changed a filter_null_values to lamda
2c6fd44 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
removed unnecassary class variable and a redundant try/except block
89540a1 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
public functions defined at interface level
52a1152 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
changed export_event and flush to private functions
3dcdcfa 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
formatting
b2714c9 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
changed connection_uuid to thread local in thrift backend
377a87b 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
made errors more specific
c9376b8 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
revert change to connection_uuid
bbfadf2 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
reverting change in close in telemetry client
9bce26b 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
JsonSerializableMixin
ef4514d 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
isdataclass check in JsonSerializableMixin
8924835 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
convert TelemetryClientFactory to module-level functions, replace Noo…
65361e7 
…pTelemetryClient class with NOOP_TELEMETRY_CLIENT singleton, updated tests accordingly

Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
renamed connection_uuid as session_id_hex
1722a77 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
added NotImplementedError to abstract class, added unit tests
e841434 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
formatting
2f89266 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
added PEP-249 link, changed NoopTelemetryClient implementation
5564bbb 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
removed unused import
1e4e8cf 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
made telemetry client close a module-level function
55b29bc 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
unit tests verbose
93bf170 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
debug logs in unit tests
45f5ccf 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
debug logs in unit tests
8ff1c1f 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
removed ABC from mixin, added try/catch block around executor shutdown
8bdd324 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
checking stuff
f99f7ea 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@github-actions GitHub Actions
Copy link

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@saishreeeee
finally
9fa5a89 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@github-actions GitHub Actions
Copy link

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@saishreeeee
remove debugging
14433c4 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee saishreeeee deployed to azure-prod June 22, 2025 11:20 — with GitHub Actions Active
@github-actions GitHub Actions
Copy link

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@saishreeeee saishreeeee marked this pull request as ready for review June 22, 2025 11:50
@github-actions GitHub Actions
Copy link

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@github-actions GitHub Actions
Copy link

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@saishreeeee
added multi threaded tests
b5bf165 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@github-actions GitHub Actions
Copy link

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@saishreeeee
formatting
307a8cc 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@github-actions GitHub Actions
Copy link

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@deeksha-db deeksha-db Awaiting requested review from deeksha-db deeksha-db is a code owner

@samikshya-db samikshya-db Awaiting requested review from samikshya-db samikshya-db is a code owner

@jprakash-db jprakash-db Awaiting requested review from jprakash-db jprakash-db is a code owner

@jackyhu-db jackyhu-db Awaiting requested review from jackyhu-db jackyhu-db is a code owner

@madhav-db madhav-db Awaiting requested review from madhav-db madhav-db is a code owner

@gopalldb gopalldb Awaiting requested review from gopalldb gopalldb is a code owner

@jayantsing-db jayantsing-db Awaiting requested review from jayantsing-db jayantsing-db is a code owner

@shivam2680 shivam2680 Awaiting requested review from shivam2680 shivam2680 is a code owner

@vikrantpuppala vikrantpuppala Awaiting requested review from vikrantpuppala vikrantpuppala is a code owner

Assignees

@saishreeeee saishreeeee

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@saishreeeee @vikrantpuppala